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SECTION  I 
INTRODUCTION 

The  Management  Information  and  Instructional  Systems  Activity  (MI ISA) 
at  Memphis  currently  provides  computer  services  to  the  Navy  Computer  Managed 
Instruction  (CMI)  system  and  to  numerous  non-CMI  users.  There  is  concern 
about  the  impact  the  non-CMI  users  are  having  on  the  ability  of  the  system 
to  meet  the  CMI  requirements  as  well  as  the  effect  of  computer  downtime  on 
training  time.  There  is  also  a  need  to  identify  other  factors  which  degrade, 
or  have  the  potential  to  degrade,  CMI  system  performance.  Accordingly,  the 
Chief  of  Naval  Education  and  Training  (CNET)  tasked  the  Training  Analysis 
and  Evaluation  Group  (TAEG)  to  identify  and  evaluate  those  factors  which 
adversely  impact  CMI  response  time.l 

PURPOSE  OF  THIS  STUDY 

The  purpose  of  this  study  was  to  provide  information  which  could  be 
used  to  maintain  system  reliabiity  as  the  CMI  processing  requirements 
increase  and  to  provide  data  to  support  expanding  the  CMI  system  capability 
to  serve  an  anticipated  increase  in  the  student  load. 

Specific  objectives  of  the  study  were  to: 

•  analyze  the  response  time,  interruptions,  and  availability  of  the 
CMI  system  for  the  period  between  March  and  October  1981 

•  identify  the  non-CMI  users,  summarize  the  requirements  these  users 
are  placing  on  the  system,  and  determine  the  impact  these  users 
are  having  on  the  ability  of  the  system  to  respond  to  CMI  require¬ 
ments 

•  identify  the  hardware/software  limitations  of  the  present  CMI 
system  and  explore  the  possibility  for  improving  both  hardware  and 
software  to  increase  efficiency,  capability,  and  reliability  of 
the  CMI  system 

•  analyze  the  relationship  between  computer  downtime  and  lost  train¬ 
ing  time  to  see  if  computer  unavailability  extends  training  time. 


APPROACH 

An  analytical  study  was  done  which  utilized  data  obtained  from  the 
following  sources: 

•  Computer  terminal  user  reports  were  collected  weekly  between  March 
to  October  1981  and  served  as  one  basis  for  assessing  the  CMI 
system  performance. 


1CNET  tasking  ltr  of  15  May  1981. 
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•  Periodic  MIISA  reports  dealing  with  system  component  failures, 
frequency  and  type  of  transactions,  non-CMI  user  requirements,  and 
processing  time  provided  the  data  from  which  conclusions  were 
drawn  about  system  capability  and  availability. 

•  Information  on  system  hardware/software  configuration  was  obtained 
from  extensive  interviews  with  MIISA  system  managers  and  support 
personnel  including  representatives  from  the  Honeywell 
Corporation. 

•  Additional  data  were  collected  from  interviews  with  school 
personnel  concerning  the  handling  of  students  during  computer 
downtime  and  problems  the  schools  were  experiencing  with  the 
central  CMI  system. 

Analyses  of  the  above  data  provided  the  basis  for  the  conclusions  and 
recommendations  of  this  study. 

ORGANIZATION  OF  THIS  REPORT 

In  addition  to  this  introduction,  this  report  contains  five  additional 
sections  and  one  appendix.  Section  II  reviews  and  summarizes  the  CMI  per¬ 
formance  data  from  March  to  October  1981.  Section  III  discusses  the 
hardware  and  software  configuration  and  limitations  of  the  present  CMI 
system.  Section  IV  discusses  the  requirements  and  problems  associated  with 
non-CMI  users  being  served  by  the  CMI  system.  Section  V  includes  an 
analysis  of  the  relationship  between  computer  downtime  and  training  time. 
Section  VI  presents  the  summary  and  recommendations.  The  appendix  contains 
detailed  information  on  various  performance  statistics. 
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SECTION  II 

ANALYSIS  OF  CMI  PERFORMANCE 
(MARCH  THROUGH  OCTOBER  1981) 

Beginning  in  March  1981  performance  statistics  were  collected  for  each 
shift  at  the  CMI  locations  (Memphis,  San  Diego,  Great  Lakes,  and  Orlando). 
Each  week  all  four  sites  forwarded  a  CMI  Computer  Terminal  Users  Report  to 
CNTECHTRA,  MUSA,  and  CNET.  Variables  for  which  data  were  collected 
included  CMI  response  time,  number  of  interruptions,  number  of  minutes  the 
system  was  down  during  the  shift,  and  the  percentage  of  time  the  CMI  system 
was  available  to  the  students.  This  section  of  the  report  summarizes  the 
performance  data  for  the  CMI  system  between  March  and  October  1981.  The 
data  for  each  variable  are  summarized  for  each  of  the  four  sites,  seven 
courses,  and  two  shifts.  An  observation  consists  of  a  measure  of  a  variable 
taken  for  a  given  day,  shift,  course,  and  location. 

RESPONSE  TIME 

The  response  time  is  measured  (in  seconds)  from  the  time  a  test  answer 
sheet  is  ejected  from  the  OPSCAN  reader  until  the  first  character  of  print 
appears  on  the  corresponding  learning  guide  at  the  terminet.  Response  time 
is  sampled  for  the  first  10  minutes  of  each  hour,  beginning  with  the  second 
hour  and  ending  with  the  sixth  hour  of  each  shift.  The  response  time  does 
not  include  the  time  required  to  print  the  learning  guide  which  varies 
depending  upon  the  length  of  the  guide.  The  procedure  for  sampling  response 
data  was  specified  by  CNTECHTRA. 

Figure  1  shows  the  monthly  distribution  of  response  times  for  all 
observations  between  March  and  October  1981.  The  average  response  time  has 
steadily  decreased  since  the  second  quarter.  By  October,  response  time  was 
averaging  less  than  three  seconds  per  transaction.  The  average  response 
time  at  Great  Lakes  is  slightly  higher  (approximately  4  seconds)  and  the 
remaining  sites  are  averaging  2  seconds  or  less  (figure  2).  Approximately 
62  percent  of  the  measured  resDOnse  times  were  3  seconds  or  less.  (See 
figure  A-l  in  the  appendix.) 

The  response  times  by  day  of  week  were  considerably  higher  for  Mondays 
during  the  March  to  June  period.  Since  then,  differences  among  the  days  of 
week  have  diminished  and  by  October  no  significant  differences  existed 
(figure  3).  Response  times  by  day  of  week  and  month  are  illustrated  in 
appendix  figure  A-2. 

Fewer  than  two  percent  of  the  observations  had  response  times  exceeding 
25  seconds,  and  fewer  than  15  percent  of  the  observations  had  times  exceed¬ 
ing  10  seconds.  Most  of  the  longer  response  times  occurred  during  the  early 
part  of  the  study  period  (figure  4). 

The  response  time  for  the  Propulsion  Engineering  (PE)  CMI  course  at 
Great  Lakes  was  slightly  slower  than  for  the  remaining  six  courses  which  are 
on  the  CMI  system  (figure  5).  This  slower  response  time  can  be  attributed 
to  differences  in  course  curriculum.  The  higher  response  time  for  the  PE 
course  causes  the  mean  response  time  at  Great  Lakes  to  be  slightly  higher 


7 


Technical  Report  119 


JON  JUL 
MONTH 


AUC  SCP  OCT 


MEMPHIS 


J 

TO  ♦ 


JUN 

MONTH 


SAN  DIEGO 


■■"ii . *u°  stp  ocT 

MONTH 

great  lakes 


12  ♦ 
! 
! 


MAR  APR  MAY  JUN 

MONTH 


ORLANDO 


Finurp  2. 


Mean  CMI  Response  Times  by  Location 


Technical  Report  119 


5  + 


4  + 


3  + 


2  + 
t 

i 

i 

i 

i  + 


Figure  3 


MON 


TUE  WED  THU  FRI 

DAY 


Mean  CMI  Response  Times  by  Day  of  Week 
(All  Observations) 


Technical  Report  119 


a, 

w 


500 


400 


300 


200 


TOO 


f 

♦ 

f 

t 

f 

; 

♦ 

t 

! 

***** 

***** 

***** 

***** 

***** 

***** 

***** 

***** 

***** 

t 

***** 

***** 

! 

***** 

***** 

♦ 

***** 

1 

***** 

***** 

! 

***** 

t 

***** 

***** 

! 

***** 

***** 

♦ 

***** 

! 

***** 

***** 

! 

***** 

***** 

! 

***** 

***** 

***** 

! 

***** 

***** 

***** 

+ 

***** 

***** 

***** 

! 

***** 

***** 

***** 

I 

***** 

***** 

***** 

! 

***** 

***** 

***** 

! 

***** 

***** 

***** 

***** 

***** 

***** 

***** 

***** 

***** 

***** 

***** 

***** 

***** 

***** 

***** 

***** 

***** 

***** 

***** 

***** 

***** 

***** 

1  2  3  4  5  10  25 


SECONDS 


Figure  4.  Histogram  of  Mean  Response  Times 
for  All  Observations 


11 


Technical  Report  119 


than  the  other  three  locations.  There  were  no  significant  differences  in 
response  times  by  day  of  week  or  shift  (appendix  figure  A-3). 

Response  time  is,  at  present,  very  fast  but  it  is  evident  that  future 
system  expansion  will  eventually  lead  to  degradation  of  system  response 
time.  Because  of  the  complex  interrelationships  of  the  system  involving 
time  of  day,  hardware  and  software  limitations,  non-CMI  user  requirements, 
and  file  maintenance  and  access  procedures,  it  is  difficult  to  ascertain, 
with  any  degree  of  confidence,  where  the  system  will  degrade.  An  actual 
scenario  or  simulation  based  scenario  must  be  analyzed  to  determine  where 
"bottlenecks"  occur  and  to  identify  factors  which  contribute  to  slow 
response  time. 

INTERRUPTIONS  AND  DOWNTIME 

The  number  of  interruptions  for  each  month  between  March  and  October 
are  shown  in  figure  6.  The  interruptions  are  sunned  for  each  location  and 
course.  When  the  CPU  is  down  at  Memphis  there  will  be  an  interruption  at 
each  course  and  location.  The  number  of  interruptions  by  day  of  week  did 
not  differ  significantly.  (See  figure  A-4  in  the  appendix.)  The  number  of 
interruptions  by  site  is  illustrated  in  figure  7. 

The  downtime  for  each  interruption  was  not  recorded  so  it  was  not 
possible  to  construct  a  precise  distribution  of  mean  downtimes.  The 
information  which  was  available  was  the  total  downtime  and  number  of  inter¬ 
ruptions  for  each  day.  The  mean  downtimes  used  in  this  study  were  estimated 
by  dividing  the  total  downtime  for  each  day  by  the  number  of  interruptions 
in  that  day.  The  mean  downtime  during  the  study  period  showed  a  general 
downward  trend.  By  the  end  of  October,  the  mean  downtime  was  estimated  to 
be  between  12  and  18  minutes  per  occurrence  (figure  8).  This  represents 
considerable  improvement  from  the  high  in  March  which  exceeded  30  minutes. 
The  mean  downtimes  by  location  are  shown  in  figure  9.  Figure  10  is  a 
histogram  of  the  downtimes  and  again  demonstrates  that  most  downtimes  were 
approximately  10  minutes  or  less.  Figure  A-5  in  the  appendix  shows  the  dis¬ 
tribution  of  downtime  duration  for  day  of  week.  No  significant  differences 
were  observed  among  the  locations  or  for  the  day  of  week. 

The  response  time,  number  of  interruptions,  and  average  duration  of 
downtime  all  interact  to  determine  the  total  time  the  CMI  system  is  unavail¬ 
able  to  the  student.  The  reported  downtime  is  intended  to  measure  or  track 
the  entire  time  during  each  shift  the  system  was  not  available  for  use. 
However,  the  method  used  for  recording  the  downtime  depends  upon  the  staff 
or  students  placing  a  demand  on  the  system,  both  when  the  system  goes  down 
and  when  it  resumes  operation.  If  there  are  no  demands  for  service  on  the 
system  when  it  goes  down,  then  the  actual  downtime  may  occur  at  some  point 
prior  to  the  point  at  which  the  observation  was  taken.  Therefore,  under 
certain  conditions  the  actual  observed  downtime  could  be  shorter  than  the 
actual  time  the  system  was  down.  Similarly,  if  the  system  resumed  operation 
and  there  was  no  demand  for  service  then  the  system  could  have  been  back  in 
operation  before  the  observation  was  taken.  This  would  tend  to  lengthen  the 
reported  downtime.  Deficiencies  in  reported  data  should,  however,  not  be 
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serious  and  the  following  data  on  total  downtime  is  submitted  recognizing 
the  above  potential  difficulty. 

The  total  downtime  shown  for  each  month  in  figure  11  is  the  total 
amount  of  time  that  CMI  services  were  not  available  for  all  courses  during 
the  month.  Downtime  could  occur  at  a  site  even  though  the  central  CPU  was 
operating.  If  the  CPU  at  Memphis  were  to  fail,  curtailing  service  to  all 
courses  and  sites,  then  the  total  downtime  as  illustrated  in  figure  11  would 
be  determined  by  adding  the  observed  downtime  for  each  course  at  each  site. 
For  example,  two  courses  at  each  of  two  sites  will  result  in  total  downtime 
of  240  minutes  when  the  CPU  goes  down  for  60  minutes.  There  was  a  signifi¬ 
cant  downward  trend  in  the  number  of  minutes  the  system  was  unavailable  from 
March  to  October  1981.  During  October  the  cumulative  downtime  for  all 
courses  at  all  sites  was  less  than  2,000  minutes  out  of  a  total  of  approxi¬ 
mately  100,000  minutes  for  the  month.  The  Basic  Electricity  and  Electronics 
(BE&E)  course  had  the  highest  number  of  minutes  of  nonavailability  because 
BE&E  is  taught  at  all  four  CMI  locations.  The  total  downtime  by  day  of  week 
and  course  is  shown  in  figures  A-6  and  A-7  in  the  appendix. 

SYSTEM  AVAILABILITY 

The  total  downtime,  measured  at  each  location,  can  result  from  any 
number  of  causes.  Although  the  entire  system  will  be  down  when  the  CPU  at 
Memphis  is  down,  there  are  other  failures  which  result  in  specific  locations 
being  down.  Figure  A-8  in  the  appendix  shows  the  downtime  for  each 
location.  Assuming  that  the  minimum  downtime  in  each  month  at  the  four 
locations  represents  the  maximum  amount  that  the  CPU  at  Memphis  could  have 
been  down,  it  is  apparent  that  failures  which  occur  at  Memphis  and  cause  the 
whole  system  to  go  down  have  been  relatively  low  during  the  latter  part  of 
the  study  period.  Actually,  the  failure  rate  of  the  CPU  will  be  signifi¬ 
cantly  lower  than  the  above  minimum  times  since  many  of  the  failures  as 
observed  at  the  site  could  be  attributed  to  site  problems.  Failure  of  the 
central  computer  was  not  a  significant  problem  during  the  study  period. 

The  percent  of  nonavai labl ity  is  computed  by  subtracting  the  number  of 
minutes  the  system  is  down  during  a  shift  from  the  total  minutes  available 
in  the  shift  during  the  day  and  dividing  the  results  by  the  total  minutes 
available  in  the  shift. 3 

Figure  12  shows  the  percent  of  time  that  the  system  was  not  available, 
averaged  for  all  sites,  during  the  period  March  to  October  1981.  Since 
March,  monthly  downtime  has  averaged  less  than  5  percent.  Total  system 
availability  was  very  high  during  the  period,  and  there  was  no  evidence  that 
system  downtime  was  a  problem.  System  availability  by  location  and  day  of 
week  at  each  location  is  presented  in  figures  A-9  and  A-10  in  the  appendix. 
System  availability  appears  to  be  slightly  higher  at  Memphis  and  Great  Lakes 
than  at  Orlando  and  San  Diego  although  the  differences  are  not  great.  There 
are  no  significant  differences  in  availability  among  the  days  of  the  week. 
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Figure  6.  Total  Interruptions  for  All  Courses 


Figure  8.  Mean  Downtimes  by  Month  for  All  Observations 
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Figure  9.  Mean  Downtimes  by  Location 
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Figure  10.  Histogram  of  Mean  Downtimes  by  Location 
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SECTION  III 

COMPUTER  MANAGED  INSTRUCTION  HARDWARE/SOFTWARE  CONFIGURATION 

The  CMI  system  hardware  configuration  is  functionally  depicted  in 
figure  13.  As  the  figure  shows,  there  are  many  subsystem  elements  involved 
in  the  processing  of  one  complete  CMI  transaction.  This  system  characteris¬ 
tic  alone  could  theoretically  lead  to  increased  downtimes  and  response  times. 
However,  this  does  not  appear  to  be  the  case  as  evidenced  by  the  statistical 
data.  Findings  indicate  that  excessive  downtime  and  degraded  response  time 
are  not  serious  problems  for  current  student  loading  levels.  It  is  not 
known,  however,  how  many  additional  students  and  courses  can  be  added  before 
unacceptable  levels  of  downtime  and  resDonse  time  will  be  experienced.  The 
elements  of  this  issue  which  relate  to  computer  hardware  and  software  con¬ 
figuration/capabilities  are  addressed  in  this  section  of  the  report. 

EQUIPMENT  HARDWARE  CAPABILITIES/LIMITATIONS 

Each  of  the  subsystem  components  shown  in  figure  13  performs  a  function 
within  the  system  and  each  is  subject  to  failure.  Consequently,  a  failure 
associated  with  data  entry,  data  communications,  multiplexing,  or  processing 
will  temporarily  cause  partial  or  total  system  failure.  The  partial  failure 
case  might  be  limited  to  an  OPSCAN  or  terminet  failure  in  which  case  only 
one  of  approximately  150  input-output  clusters  would  be  affected.  This  type 
of  failure  would  normally  be  corrected  in  10  to  15  minutes  by  replacing  the 
failed  unit  with  a  spare.  Then  the  failed  unit  would  be  repaired  to  main¬ 
tain  a  backup  capability.  A  total  site  failure  could  be  caused  by  an  inter¬ 
ruption  in  communications  which  results  in  network  separation  between  the 
site  concentrator  (Honeywell  Level  6)  and  the  central  processors  (Honeywell 
series  60  dual  processor)  at  Memphis.  If  other  operable  communications 
circuits  could  not  be  used,  this  failure  would  remain  for  the  duration  of 
the  communications  line  failure.  However,  it  would  only  affect  operations 
at  the  site  losing  communications.  Additionally,  the  probability  of  this 
occurring  is  reasonably  low.  The  most  serious  failure  would  he  one  in  which 
common  elements  affecting  the  dual  central  processors  would  fail.  This 
might  be  due  to  power  failure,  fire,  flood,  or  some  other  similar  serious 
occurrence.  For  a  case  of  this  nature,  conceivably  the  CMI  system  might  be 
down  for  days  or  weeks.  Although  this  type  of  failure  is  not  very  likely, 
an  occurrence  would  impact  a  significant  portion  of  total  Navy  training.  It 
is  for  a  potentiality  such  as  this  that  a  CMI  manual  backup  system  should 
always  be  available.  For  networks  utilizing  a  single  central  processing 
site,  high  reliability  and  reasonable  redundancy  (both  of  which  have  been 
designed  into  the  existing  CMI  system)  can  only  decrease  the  risk  of  total 
system  failure.  Only  distributed  autonomous  transaction  processing  will 
assure  that  total  CMI  system  failure  (all  subsystems  inoperable  concurrently) 
will  not  occur. 

The  following  discussion  will  be  directed  toward  specific  network  sub¬ 
system  elements  in  order  to  provide  a  more  detailed  assessment  of  system 
capabilities/limitations. 
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Figure  13.  CMI  Svstem  Hardware  Configuration 
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INPUT-OUTPUT  LEARNIN6  CENTER  CLUSTERS.  The  input-output  learning  center 
clusters  can  be  used  to  input  student  tests,  each  having  up  to  50  multiple 
choice  responses,  through  an  OPSCAN-17  optical  mark  reader  or  to  output  learn¬ 
ing  guides  on  a  Terminet  1200  keyboard/print  terminal.  This  input-output 
channel  can  also  be  used  to  enter  administrative  transactions  and  to  receive 
administrative  responses.  Input-output  control  logic  for  these  clusters  is 
contained  in  the  base  of  the  OPSCAN  unit.  A  communications  interface  to  the 
Honeywell  Level  6  concentrator  is  established  via  G0C-202-9B  modems. 

Because  the  OPSCAN  reader  and  the  terminet  printer  are  electro-mechanical, 
failure  rates  for  these  devices  are  higher  than  for  electronic  subsystem 
elements.  These  units  do  have  a  relatively  high  failure  rate  but  spares  are 
normally  kept  on  hand,  if  available,  to  replace  failed  units.  By  using  this 
maintenance  strategy,  cluster  failures  at  sites  with  backup  units  do  not 
objectionably  degrade  system  capability  or  availability.  It  does  appear, 
however,  that  locations  without  a  backup  capability  could  seriously  degrade 
system  availability.  The  control  logic  and  modems  have  proved  to  be  highly 
reliable  and  the  communications  lines  which  connect  the  clusters  to  the 
concentrator  are  similarly  reliable.  This  input-output  configuration, 
although  not  the  best  or  most  reliable  by  today's  standards,  provides  an 
adequate  system  capability.  Many  improvements  are  possible  for  this  network 
node  such  as  hand-held  device  input,  higher  speed  input,  higher  speed  printer 
output  and  keyboard/display  testing  terminals.  However,  the  present  configura¬ 
tion  provides  satisfactory  performance  and  any  proposed  improvements  would  have 
to  be  individually  assessed  on  the  basis  of  cost-benefit  projections. 

HONEYWELL  LEVEL  6  CONCENTRATORS..  The  Honeywell  Level  6  concentrators  are  in 
essence  communications  computers  which  multiplex  inputs  from  the  learning 
center  clusters  for  transmission  to  the  central  computer  complex  in  Memphis. 
Return  data  in  the  form  of  learning  guides  or  administrative  responses  are 
also  routed  to  the  proper  receiving  station  via  this  subsystem.  Since  the 
data  rate  on  outgoing  or  incoming  communication  lines  is  relatively  low  in 
terms  of  computer  capability,  response  time  should  not  be  limited.  In  addi¬ 
tion,  the  reliability  of  this  equipment  has  proven  to  meet  or  exceed  expecta¬ 
tions.  Although  other  processing,  beyond  that  required  for  communications, 
is  accomplished  within  this  computer,  only  a  failure  affecting  multiplexing 
or  communications  would  impact  student  testing.  Because  all  sites  do  not 
have  subsystem  redundancy,  it  is  possible  that  failure  could  cause  CMI  inter¬ 
ruptions  for  extended  periods  (many  hours).  Failure  data  collected  for  the 
past  6  months,  however,  do  not  show  this  to  be  a  problem  and,  in  the  opinion 
of  experienced  personnel,  continued  high  reliability  is  expected. 

FRONT-END  COIWUNICATIONS  PROCESSORS.  The  network  arrangement  between  the 
student  input-output  clusters  and  the  concentrator  is  repeated  in  concept  at 
the  Memphis  host  computer  site.  For  this  application  a  front-end  communi¬ 
cations  processor  is  utilized  to  multiplex  inputs  from  many  site  locations. 

This  processor  acts  as  a  temporary  buffer  and  switch  directing  incoming 
transactions  to  buffer  locations  in  main  processor  memory.  This  provides 
temporary  transaction  data/storage  while  awaiting  central  processor  service. 
When  service  is  completed,  the  transaction  response  information  is  routed  to 
the  proper  output  communications  channel  via  the  communications  processor. 
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Discussions  with  system  personnel  have  indicated  that  the  memory  limitation 
on  the  front-end  processors  sometimes  results  in  system  overload  and  tem¬ 
porary  failure.  These  failures  are  not  normally  catastrophic  since  a  system 
reboot  will  return  the  system  to  operational  status  in  a  period  of  minutes. 
It  is  a  problem,  however,  which  should  be  diagnosed  further  to  determine 
what  corrective  measures  should  be  considered. 

HONEYWELL  SERIES  60  DUAL  PROCESSORS  AND  PERIPHERALS.  The  equipment  at  the 
Memphis  central  CMI  processing  center  consists  of  2  Honeywell  series  60 
processors,  23  100-MB  disk  drives,  11  magnetic  tape  units,  4  front-end 
communications  processors,  2  1200  LPM  printers,  a  card  reader,  and  a  card 
punch.  The  processors  share  one  mega  word  of  memory  and  are  configured  to 
service  multiple  users.  The  multi-tasking,  multi-processing  operating 
system  combined  with  the  data  base  management  and  input-output  handling 
systems  combine  to  provide  a  powerful  and  effective  computational  complex. 
Although  the  CMI  system  uses  only  a  portion  of  the  total  resources  available 
(for  example,  CMI  uses  only  three  of  the  23  disk  drives  available),  it  is 
the  highest  priority  user  and  is  not  affected  by  other  users  in  terms  of 
response  time.  A  system  crash  could  result  from  defective  non-CMI  applica¬ 
tions  software  or  front-end  processor  overload.  However,  this  possibility 
is  not  considered  serious  because  most  recorded  failures  in  this  category 
have  been  corrected  within  reasonable  time  limits.  The  peripherals  have 
also  proven  to  be  reliable  and  of  sufficient  capacity  to  handle  peak 
loading.  An  analysis  of  system  capabilities  has  shown  that  there  are  some 
CMI  software  characteristics  which  would  limit  system  expansion.  However, 
it  appears  that  these  characteristic  limitations  could  be  corrected  with 
some  system  redesign.  This  topic  is  discussed  further  in  the  following 
paragraph. 

SOFTWARE  CAPABILITIES  AND  LIMITATIONS 

The  system  resident  software  in  the  CMI  network  consists  of  operating 
systems,  data  base  management  systems,  communications  programs,  and  utility 
programs.  This  software,  with  a  few  minor  exceptions,  has  proven  to  be 
reliable.  Some  relatively  minor  modifications  have  been  made  to  the 
resident  vendor  developed  software  for  special  applications. 

The  CMI  software  which  controls  transaction  processing  is  considered  to 
be  of  primary  importance.  The  software  consists  primarily  of  an  evaluation 
program,  which  provides  the  test  evaluation  and  learning  guide  generation 
capability,  and  a  number  of  administrative  transaction  processing  programs. 
Although  the  administrative  programs  such  as  registration,  class  rosters, 
student  progress,  and  those  concerning  student  flow  are  necessary  for 
effective  school  management,  normally  they  are  not  competing  for  time  with 
evaluation.  They  are  considered  administrative  batch  operations  and  are 
normally  scheduled  for  minimum  impact  on  student  testing. 

The  evaluation  program  which  controls  the  analysis  of,  and  response  to, 
student  tests  is  the  primary  response  time  controlling  program.  At  present, 
the  maximum  test  transaction  throughput  rate  is  approximately  two  trans¬ 
actions  per  second  (120  per  minute  or  7,200  per  hour).  If  it  were  possible 
to  maintain  this  rate  for  two  six-hour  shifts,  which  is  unlikely,  86,400 
test  transactions  could  be  serviced.  This  assumes  that  no  administrative 
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transactions  are  competing  for  time  and  that  all  answer  sheets  are  properly 
coded.  It  also  assumes  a  continuous  flow  of  test  input.  For  the  measured 
case,  approximately  35,000  test  transactions  are  serviced  each  day  with  the 
balance  of  the  time  being  utilized  for  administrative  transactions  and  error 
transactions  with  periods  of  nonutilization  interspersed  throughout  the  day. 
For  the  current  student  loading  of  approximately  8,500,  the  existing  evalua¬ 
tion  capability  is  adequate.  Making  changes  to  the  evaluation  program  to 
allow  for  multi-tasking  operation  within  the  evaluation  program  would  have 
the  effect  of  handling  many  test  transactions  at  a  time  as  opposed  to  one  at 
a  time  for  the  present  case. 

Upgrading  the  evaluation  program  appears  to  be  a  relatively  simple 
solution  to  satisfy  a  potential  need  for  greatly  increasing  throughput. 
However,  for  a  change  of  this  nature,  careful  study  should  precede  develop¬ 
ment.  This  may  not  be  a  satisfactory  solution  unless  significant  disk  and 
file  restructuring  accompany  the  multi-tasking  approach.  If  a  single  disk 
access  to  a  course  file  locks  that  file  out  for  a  following  transaction,  a 
wait  period  of  20  to  50  milliseconds  or  more  might  result  which  could 
partially  negate  the  expected  benefit  of  multi-tasking.  The  net  result  of 
this  situation  might  be  a  moderate  improvement  in  throughput  and  not  the 
significant  improvement  expected.  File  reorganization,  and  new  approaches 
to  course  file  development,  if  effectively  accomplished,  together  with 
multi-tasking,  should  minimize  the  number  of  disk  accesses  and  improve  the 
overall  response  time.  By  following  this  approach,  high  priority  file 
handling  could  be  limited  to  no  more  than  two  disk  accesses  as  compared  to 
the  present  5  to  12.  This  would  then  be  followed  by  housekeeping  trans¬ 
actions  which  would  be  accomplished  in  background  mode. 

Even  if  these  improvements  in  the  evaluation  were  accomplished,  other 
hardware  and  software  response  time  constraints  might  occur.  These  con¬ 
straints  might  be  due  to  data  net  overload,  student  cluster  overload,  or 
inadequate  buffer  storage.  It  is  suggested  that  a  total  system  network 
analysis  be  done  to  determine  the  maximum  throughput  at  each  node  before  any 
single  measure  is  taken  to  improve  system  performance.  One  obvious  approach 
to  system  expansion,  if  required,  would  be  to  provide  a  distributed  process¬ 
ing  capability  at  each  site.  Although  this  approach  has  certain  detrimental 
effects  which  would  offset  some  of  the  benefits  to  be  gained,  it  appears  to 
be  a  reasonable  expansion  option.  This  issue  is  discussed  in  the  following 
paragraph. 

DISTRIBUTED  PROCESSING  CONSIDERATIONS 

The  phrase  "distributed  processing"  is  often  suggested  as  a  solution  to 
many  of  today's  data  processing  problems.  Whether  the  problem  is  response 
time,  insufficient  storage  capacity,  or  data  ba.e  management  inadequacy, 
there  is  a  tendency  to  favor  a  corrective  measure  in  the  form  of  distributed 
processing.  This  assessment  of  distributed  processing  stresses  the  need  for 
cautious  evolutionary  development  when  considering  this  alternative  for  CMI 
system  expansion. 

It  appears  inevitable  that  distributed  processing  will  play  a  signifi¬ 
cant  role  in  the  future  and  could  provide  the  means  for  greatly  increasing 
current  system  capacity.  However,  it  should  not  be  considered  a  potential 
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cure  all.  The  primary  question  which  should  be  addressed  when  assessing 
this  option  is,  how  many  and  which  processing  functions  can  be  efficiently 
distributed.  There  are  also  questions  concerning  loose  or  tight  coupling 
within  the  network,  the  necessity  for  distributed  data  base  management,  com¬ 
munications  protocol,  privacy/security/integrity,  and  network 
management/control.  Dispersing  many  CMI  processing,  storage,  and  reporting 
^unctions  without  maintaining  strong  and  effective  centralized  policy 
development  and  management  control  could  bring  about  a  degradation  in  CMI 
system  performance  instead  of  the  desired  improvement. 

The  distributed  approach  has  many  advantages.  It  would  not  be 
necessary  to  communicate  student  test  response  data  hundreds  or  thousands  of 
miles  for  response  analysis  and  learning  guide  generation  as  is  now  the 
case.  Test  transaction  processing  is  well  within  the  capability  of  medium 
scale  computers  which  could  be  located  at  remote  sites.  A  remote  computer 
of  this  type  could  also  provide  the  processing  functions  associated  with 
class  roster  generation,  predicted  completion  time  (PCT)  resource 
allocation/scheduling,  and  site  level  administrative  support.  However,  it 
might  not  be  in  the  best  interest  of  the  Naval  Education  and  Training 
Command  to  distribute  such  functions  as  student  registration,  student  record 
keeping,  student  tracking,  and  training  pipeline  management.  These  examples 
are  not  offered  as  recommendations.  They  only  demonstrate  the  extent  of  the 
analysis  required  before  making  hard  decisions  relating  to  CMI  configuration 
changes.  They  also  demonstrate  the  necessity  for  an  in-depth  analysis  of 
CMI  long-ranqe  requirements  before  selecting  a  course  of  action  for  system 
redesign. 
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SECTION  IV 
HOH-CMl  USERS 

The  Honeywell  6000  computer  system,  located  at  Naval  Air  Technical 
Training  Center  (NATTC),  Millington,  is  used  for  more  than  CMI  support  (see 
figures  14  and  15).  This  system  supports  numerous  other  functions  including 
naval  technical  training,  recruit  training,  and  miscellaneous  activities. 

The  largest  single  user  (where  use  is  measured  by  processing  time)  is 
the  Military  Personnel  Information  System  (MILPERS IS).  Information  is 
passed  to  MILPERSI S  throughout  the  day  to  update  various  data  concerning 
students  within  CNTECHTRA.  Intense  MILPERSI S  use  of  the  Honeywell  6000 
system  is  reserved  for  the  1800-2300  Central  Standard  Time  period  when  CMI 
use  is  very  low. 

The  second  and  third  largest  users  of  processing  time  are  CMI  and  the 
maintenance  of  the  MIISA  General  Computer  Operating  System  (GCOS). 

Together,  these  three  large  users--MILPERSIS,  CMI,  and  MIISA  operating 
systems  maintenance— account  for  approximately  85  percent  of  the  Honeywell 
6000  processing  time.  The  remaining  15  percent  is  used  for  numerous 
functions,  including  primarily: 

•  Standard  Transfer  Directive  Module  (STDM)  for  ordering  the  trans¬ 
fer  of  students 

•  NATTC,  Millington,  civilian  and  military  payrolls,  and  civilian 
personnel  support 

•  NATTC,  Millington,  logistical  functions;  e.g..  Navy  Stock  Fund  and 
Resource  Management  System 

•  U.S.  Army  Corps  of  Engineers  support 

•  Individual  Flight  Activity  Report  Subsystem  (IFARS)  for  managing 
flight  personnel.  The  Honeywell  6000  relays  IFARS  data  from 
Memphis  to  Pensacola  for  central  processing. 

•  Surface  Warfare  Officer  School  support 

•  Availability  Reporting  and  Tracking  Module  (ARTM)  and  Recruit 
Accession  Module  (RAM)  to  aid  in  personnel  management  within  the 
Recruit  and  Student  Training  Commands.  Although  this  support  is 
provided  primarily  on  Level  6  systems,  the  Honeywell  6000  does 
provide  some  central  processing  support. 

•  Naval  Air  Maintenance  Training  Group  and  Air  Maintenance  Detach¬ 
ment  support 

•  Computer  Driven  Training  System  which  provides  CAI-like  instruc¬ 
tion  in  computer  use. 


29 


90- 


-9-**-**i 


(%)  3WU  DNISS300bd  IVlOi 


Technical  Report  119 


30 


Figure  14.  Distribution  of  Honeywell  6000  Processing  Time  (%)  Among  Users 
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Figure  15.  Distribution  of  Honeywell  6000  Processing  Time  (Hrs)  Among  Users 
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Currently,  there  is  no  statistical  basis  for  relating  degraded  CMI 
performance  to  non-CMI  user  applications.  Performance  indicators  reviewed 
show  typical  CMI  response  times  of  3  to  5  seconds  and  overall  system 
availability  exceeding  95  percent.  In  addition,  there  is  no  indication  that 
non-CMI  users  cause  more  than  a  proportionate  number  of  failures  of  those 
recorded.  It  is  obvious  that  non-CMI  use  will  increase  the  probability  of 
total  system  failure,  but  it  can  not  be  determined  at  the  present  time  if 
this  increased  risk  of  failure  would  warrant  a  reduction  in  service  to  non- 
CMI  users.  By  allowing  multiple  users,  system  utilization  is  increased  and 
the  return  on  computer  investment  is  positively  affected.  Although  there 
are  a  number  of  ways  to  improve  the  operational  availability  of  the  CMI 
system,  as  discussed  in  section  III  of  this  report,  it  appears  doubtful  that 
limiting  non-CMI  use  beyond  current  levels  would  have  any  significant 
impact. 
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SECTION  V 

RELATIONSHIP  BETWEEN  CMI  DOWNTIME 
AND  STUDENT  TRAINING  TIME 

IMPACT  OF  COMPUTER  DOWNTIME 

The  most  obvious  and  potential  impact  of  CMI  downtime  on  the  cost  of 
training  arises  from  extending  the  time  required  for  students  to  complete 
training.  An  objective  determination  of  how  CMI  downtime  affects  training 
time  requires  the  collection  of  data  on  student  activity  during  downtime 
and  the  use  of  empirical  measures  of  training  time  as  a  function  of  CMI 
availability.  Such  data  were  not  collected  during  past  CMI  downtimes  and 
during  the  period  covered  by  this  study  availability  was  so  high  that  there 
was  an  insufficient  number  of  adverse  effects  from  which  to  deduce  a  func¬ 
tional  relationship  between  downtime  and  training  time. 

At  least  two  important  factors  contribute  to  the  relationship  between 
training  time  and  CMI  downtime.  The  first  arises  from  the  role  of  CMI  in 
the  instructional  process.  The  Navy  CMI  system  itself  is  not  designed  to 
provide  instruction  during  the  time  the  student  interacts  with  the  computer. 
Apart  from  the  incidental  learning  which  takes  place  during  tests,  the 
majority  of  instruction  occurs  while  the  student  is  in  the  carrel  and  not 
interacting  with  the  computer.  The  computer  provides  periodic  performance 
evaluations  and  directs  the  student  in  future  study  assignments  by  issuing 
learning  guides.  Consequently,  when  the  computer  is  down  those  students  who 
are  not  ready  for  a  performance  evaluation  will  not  be  directly  affected. 

If  the  downtime  interval  is  very  short  (as  most  have  been  in  the  last  four 
months)  and  the  frequency  of  student  interaction  is  low,  then  very  few  stu¬ 
dents  would  even  be  aware  that  the  CMI  system  was  down  and  there  would  be  no 
impact  on  those  students.  For  those  students  who  were  affected,  the  average 
waiting  time  for  the  computer  would  be  only  a  fraction  of  the  computer  down¬ 
time,  assuming  students  demand  service  at  a  constant  and  uniform  rate  during 
the  shift. 

The  second  factor  which  impacts  on  the  relationship  between  training 
time  and  CMI  downtime  is  how  the  student  is  managed  in  the  classroom  once 
the  student  demands  service  and  finds  the  computer  down.  Arguments  are  fre¬ 
quently  advanced  that  the  amount  of  lost  training  time  for  any  student  can 
be  measured  from  the  time  the  student  demands  service  and  finds  it  unavail¬ 
able  to  the  point  he/she  obtains  the  requested  service.  This  argument  must 
be  predicated  on  the  assumption  that  learning  stops  when  required  CMI 
service  is  not  available.  Since  the  computer  plays  its  role  in  the  manage¬ 
ment  of  instruction,  and  not  the  instruction  itself,  such  an  assumption  is 
untenable. 

One  alternative  for  obtaining  data  from  which  to  derive  the  relation¬ 
ship  between  training  time  and  computer  downtime  would  be  to  devise  an 
experiment  in  which  data  on  student  queues,  training  time,  and  effectiveness 
would  be  compiled  and  analyzed  following  controlled  CMI  shutdowns.  Such  an 
experiment  was  considered  not  to  be  feasible  with  the  operational  CMI 
system. 
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An  alternative  to  the  experimental  approach  is  to  perform  a  logical 
analysis  of  the  problem  using  gualitative  data  drawn  from  previous  research 
dealing  with  CMI  systems  and  discussions  with  school  management  as  to  how 
current  CMI  shutdowns  are  handled.  The  following  discussion  is  based  UDon 
an  analysis  of  those  factors  which  determine  the  training  time  reguired  by 
students,  an  analysis  of  how  the  computer/CMI  system  interacts  with  those 
factors  which  determine  t  aining  time,  and  an  assessment  of  the  management 
of  students  at  the  schools  when  the  CMI  system  is  down. 

FACTORS  INFLUENCING  STUDENT  TRAINING  TIME 

A  review  of  training  literature  shows  that  "student  training  time"  has 
many  components.  Carroll  (1963)  identifies  five  factors  which  interact  to 
determine  the  total  student  training  time.  These  include:  (1)  time  allowed 
for  learning,  (2)  time  the  learner  is  willing  to  spend,  (3)  time  reguired 
because  of  the  student's  ability,  (4)  ability  to  understand  instruction,  and 
(5)  guality  of  instruction.  Bloom  (1976)  has  demonstrated  that  through 
individualizing  instruction,  considering  each  student's  unigue  status  with 
respect  to  the  above  five  factors,  certain  conditions  of  learning  can  be 
established  which  facilitate  the  student  in  learning.  The  importance  of 
these  conditions  for  the  present  problem  is  evident  in  the  statement,  "... 
what  any  person  in  the  world  can  learn  almost  all  persons  can  learn  if 
provided  with  appropriate  prior  and  current  conditions  of  learning"  (Bloom, 
1976,  p.  7).  When  these  conditions  of  learning  are  optimum,  we  can  get 
almost  anyone  to  learn  almost  anything.  There  are  six  conditions: 

1.  Prerequisites  (PRQ).  These  are  the  knowledge  and  attitudes  that 
students  bring  to  the  learning  situation  based  on  previous  experience. 

ASVAB  scores  and  reading  and  computational  scores  are  often  used  to  assess 
this  condition.  Attitudes  are  reflected  in  measures  of  motivation  and 
perseverance.  The  best  adaptive  instruction  accommodates  student  variation 
among  the  prerequisites. 

2.  Cues  (CUE).  This  condition  involves  the  ways  the  instructor 
informs  students  what  they  are  to  do,  and  includes  learning  objectives, 
verbal  explanations,  demonstrations,  and  models. 

3.  Participation  (PAR).  In  order  to  learn,  the  student  must  overtly 
or  covertly  do  something.  Both  instructors  and  the  instructional  materials 
must  keep  the  student's  mind  intently  engaged  with  the  subject  matter.  A 
very  high  relationship  exists  between  such  mind  engagement  and  amount 
learned.  Such  "mind  engagement"  time  is  not  necessarily  related  to  the 
amount  of  time  a  student  spends  in  the  carrel.  PAR  is  the  study  time  that 
remains  after  subtracting  wasted  time  from  the  time  a  student  spends  in  a 
learning  center  or  laboratory. 

4.  Reinforcement  (RNF).  A  reinforcer  is  part  of  the  reward  or 
motivational  system  that  strengthens  the  behavior  that  precedes  its 
administration.  The  definition  is  circular  in  that  if  the  behavior  is  not 
strengthened,  whatever  was  administered  was  not  a  reinforcer.  In  the 
science  of  instruction,  here  is  where  the  "art  of  teaching"  comes  in.  There 
can  be  no  formula  for  the  use  of  reinforcers.  It  takes  a  wise  and  sensitive 
instructor  to  know  when  to  use  external  reinforcers  such  as  recognition  or 
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privilege  and  internal  reinforcers  such  as  leaving  the  student  alone  when 
the  instructional  materials  are  obviously  strengthening  PAR. 

5.  Feedback  (FBK).  This  is  the  information  that  informs  students  of 
the  degree  to  which  their  practice  is  discrepant  from  that  which  they  are 
supposed  to  be  practicing.  Sometimes  FBK  and  RNF  are  the  same;  other  times 
FBK  is  neutral.  Tests,  critiques,  and  oral  examinations  are  used  for  this 
function  in  an  individualized  instructional  system. 

6.  Correctives  (COR).  After  the  FBK  shows  a  discrepancy  between  the 
required  and  the  demonstrated  response,  the  corrective  prescribes  some  sort 
of  learner  activity  that  will  eliminate  the  discrepancy. 

Wherever  there  is  efficient  and  effective  instruction,  individualized 
or  lock-step,  these  six  conditions  of  learning  are  present. 

A  LOGICAL  ANALYSIS  OF  QUEUES  AND  INTERRUPTIONS  IN  THE  NAVY  CMI  SYSTEM 

The  Navy  CMI  system  is  designed  to  assist  primarily  in  two  of  the  above 
six  conditions,  namely  feedback  and  corrective  functions.  The  CMI  program 
is  tailored  to  assist  the  learning  center  instructor  (LCI)  in  providing 
alternative  forms  of  tests,  retakes  of  examinations,  and  prescriptions  for 
corrective  actions  for  student  weaknesses.  Information  contained  in  the  CMI 
system  Student  Progress  Reports  also  serves  the  prerequisite  and  partici¬ 
pation  functions.  If  a  student  progresses  through  one  or  two  instructional 
modules  every  day,  takes  30  minutes  to  take  a  test  and  score  it  at  the 
OPSCAN  terminal,  and  demonstrates  mastery  on  about  the  second  attempt,  a 
conservative  estimate  of  the  proportion  of  his  or  her  total  learning  center 
time  spent  in  test  taking  would  average  approximately  one  hour  per  day. 

Even  during  this  period,  it  is  estimated  that  the  student  would  interact 
with  the  computer  for  less  than  five  minutes  of  the  time.  When  the  CMI 
system  is  down,  learning  need  not  stop,  although  the  sequence  in  which  the 
student  undertakes  the  learning  experience  is  usually  modified. 

The  flow  diagram  in  figure  17  is  based  on  how  a  CMI  learning  center 
activity  might  operate  when  the  computer  is  down.  There  are  five  key  po  rts 
in  the  model  where  student  queues  might  be  expected  to  develop  following  an 
extended  period  of  computer  downtime.  They  are  awaiting: 

1.  the  first  lesson  or  module  assignment  in  the  course, 

2.  instructor  help  or  approval  to  attempt  a  formative  or  module 
examination, 

3.  examination  scoring  and  study  assignment  at  the  OP SCAN  terminal, 

4.  instructor  diagnosis,  counsel,  and  prescription  following  failure 
to  demonstrate  mastery  on  a  test,  and 

5.  assignment  or  materials  for  the  next  lesson. 

These  points  are  noted  as  QUEUE1  through  QUEUES,  respectively,  in  the  flow 
diagram.  As  the  diagram  shows,  four  of  these  possible  delays  are  directly 
related  to  the  LCI's  availability  of  time. 
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Following  a  shift  to  the  manual  mode  because  of  a  computer  failure,  the 
LCI  must  become  occupied  with  keeping  the  students  productive.  This  involves 
keeping  them  actively  pursuing  course  goals  and  manually  entering  updated 
student  information  into  the  records  system.  Perfectly  designed  instruc¬ 
tional  materials  would  support  the  important  conditions  of  learning  so  tha* 
the  instructor  would  be  free  to  carry  the  computer's  share  of  the  management. 
Since  the  instructional  materials  are  seldom  so  designed,  it  is  reasoned 
that  the  instructor  would  eventually  become  involved  in  all  phases  of  the 
instruction  and  such  manual  management  activity  will  begin  competing  with 
the  students'  needs  for  cues,  feedback,  reinforcement,  and  correctives. 

Many  factors  would  determine  how  long  the  computer  must  be  down  before  this 
overload  would  become  a  serious  problem.  It  is  estimated  that  on  the  average 
it  would  not  become  a  serious  problem  for  downtimes  under  1  hour. 

MANAGEMENT  OF  STUDENTS  OURING  COMPUTER  DOWNTIME 

Figure  17  illustrates  how  instructors  might  intervene  at  each  QUEUE 
position  should  the  computer  system  fail.  Presently,  instructors  do  not 
usually  intervene  (at  QUEUE2)  to  determine,  by  oral  examination,  that  the 
student  is  ready  for  the  test  and  has  a  high  probability  of  being  successful 
--but  if  they  were  required  to  go  into  a  manual  mode  this  intervention  would 
be  necessary.  Presently,  the  QUEUE3  requires  a  short  wait  for  scoring  and 
test  results— but  in  a  manual  mode,  this  wait  would  certainly  be  longer. 
Instructors  do  not  necessarily  override  the  study  assignment  accompanying 
the  printout  of  test  results  in  order  to  give  a  personal  diagnosis  of  learn¬ 
ing  difficulties— but  they  can,  and  in  a  manual  mode  they  would  provide  the 
personal  diagnosis. 

A  hypothetical  example  of  how  a  competent  LCI  who  knows  his  or  her 
students  should  manage  computer  downtime  would  be  as  follows: 

Students  who  are  known  to  have  conceptual  difficulties  with  the  pre¬ 
requisite  learning  modules  and  are  running  *ar  behind  their  predicted  com¬ 
pletion  time  (PCT)  are  requested  to  spend  additional  time  studying  in  their 
present  module.  Such  study  could  be  in  an  additional  method  of  presenta¬ 
tion,  such  as  the  summary,  narrative,  or  programmed  instruction  mentioned  in 
NAVEDTRA  110A.  It  could  be  in  the  form  of  an  elaboration  of  the  module  goals 
by  film,  filmstrip,  trip  to  the  lab,  or  peer  instruction  by  an  advanced  stu¬ 
dent.  The  goal  of  such  activity  would  be  for  the  student  to  demonstrate 
mastery  of  the  present  module  on  the  first  attempt.  Such  "overlearning"  is 
not  necessarily  inefficient  for  this  student;  it  may  be  the  learning-how-to- 
learn,  or  the  confidence-builder,  that  will  cause  him  or  her  to  "takeoff" 
after  the  CMI  system  comes  back  on  line. 

For  students  who  are  ahead  of  their  PCT  and  who  generally  demonstrate 
mastery  of  modules  on  the  first  test,  the  instructor  would  present  subsequent 
modules.  The  time  required  for  testing  seems  to  slow  these  students'  progress. 
They  study  in  the  learning  center  or  lab  for  the  knowledge,  not  just  to  pass 
tests.  Such  students  can  go  back  and  successfully  take  several  module  tests 
without  any  debilitating  effects  long  after  the  computer  comes  back  on  line. 

For  the  majority  of  students  between  these  two  extremes,  a  period  of 
computer  downtime  will  require  an  LCI  who  has  the  ability  to  differentiate 
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amonq  students  and  manually  make  assignments.  In  order  to  ensure  that  the 
students  make  effective  and  efficient  use  of  their  time,  the  assignments 
must  be  based  on  the  students'  progress  with  respect  to  their  PCT,  their 
motivation,  and  their  self-reliance.  Instructors  must  be  able  to 
effectively  utilize  audio-visual  aids  and  other  materials  which  are 
available  to  ensure  student  progress.  The  management  of  the  student  during 
computer  downtime  provides  an  opportunity  for  LCIs  to  demonstrate  how 
essential  they  are  in  facilitating  the  conditions  of  learning.  It  is  not 
easy,  and  periods  of  downtime  are  not  without  stress,  but  instructors  should 
be  selected  who  can  handle  problems  which  arise  during  periods  when  the 
computer  temporarily  fails. 

EVIDENCE  OF  THE  IMPACT  OF  COMPUTER  DOWNTIME  ON  TRAINING  TIME 

There  is  little  empirical  data  demonstrating  the  effect  of  computer 
downtime  on  training.  The  evidence  which  does  exist  seems  to  suggest  that 
computer  downtime  need  not  halt  student  progress. 

Several  studies  (Judd,  McCombs,  and  Dobrovolny  (1979);  Diamond  (1969); 
Ammentorp,  Morris,  and  Miller  (1973))  have  demonstrated,  in  general,  that 
even  though  the  CMI/CAI  systems  they  were  studying  had  interruptions, 
students  could  still  maintain  acceptable  progress  if  their  time  were 
adequately  managed.  Difficulties  which  had  to  be  overcome  included  the  need 
to  provide  adequate  student  feedback  and  student  time  management.  In 
general,  the  manual  management  methods  were  less  efficient  than  the 
computer. 

Two  Navy  CMI  sites  (Memphis  and  Orlando)  were  visited  to  obtain  data  on 
the  effect  of  computer  downtime  on  training.  The  following  questions  were 
posed  to  school  staff  personnel  who  have  had  experience  with  the  CMI  system: 

1.  How  often  does  the  computer  system  go  down? 

2.  About  how  long  does  the  computer  usually  stay  down? 

3.  What  do  learning  center  people  do  when  the  computer  is  down? 

4.  Do  you  have  a  manual  or  back-up  system? 

5.  How  long  are  learning  center  queues  per  downtime? 

6.  How  long  must  the  system  be  down  to  affect  student  time  to 
mastery? 

Individuals  who  responded  to  the  above  questions  were  generally 
cognizant  of  problems  which  existed  before  March  1 981 — the  last  time  there 
were  any  appreciable  downtime  problems.  There  are  several  generalizations 
to  be  drawn  from  responses  to  these  six  questions.  First,  there  have  been 
very  few  problems  since  March  1981  in  terms  of  response  times  or  inter¬ 
ruptions.  When  there  were  interruptions,  they  have  not  lasted  for  more  than 
a  few  minutes.  Second,  most  learning  centers  do  not  have  a  paper  and  pencil 
back-up  system  for  testing,  so  during  downtime  students  do  additional  study. 
If  the  students  are  taking  a  test,  they  just  review  their  work  a  little 
longer  before  submitting  it  to  the  OPSCAN.  If  the  downtime  is  long,  the  LCI 
assigns  the  students  the  next  instructional  module  in  their  learning 
sequence.  Finally,  queues  following  the  return  of  the  computer  are  usually 
very  short,  althouqh  there  were  examples  of  nearly  2-hour  queues  which 
occurred  over  a  year  ago. 
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The  question  of  how  long  the  system  must  be  down  before  there  would  be 
an  adverse  effect  on  training  time  produced  responses  which  varied  from  15 
minutes  to  1  hour.  Much  of  the  variation  in  these  estimates  could  be 
attributed  to  how  well  the  school  was  staffed,  prepared,  or  had  available 
back-up  procedures. 

The  cost  of  computer  downtime  can  be  computed  under  two  alternative 
assumptions.  The  first  assumption  (Case  I)  would  represent  an  upper  limit 
to  the  cost  of  computer  downtime,  except  perhaps  for  extremely  long 
downtimes  of  several  days  during  which  the  entire  training  program  might 
need  to  be  restructured.  Case  I  assumes  that  for  each  minute  a  student 
spends  waiting  for  the  computer  that  training  time  is  extended  on  a  one  for 
one  basis.  The  formula  for  computing  lost  training  time  for  Case  I  is  as 
follows : 

L  =  ( 1/2 ) ( C) ( S) 

where:  L  =  Lost  Training  Hours 

C  =  Hours  Computer  was  down 
S  =  Students  demanding  service  per  hour 

The  second  assumption  (Case  II)  is  based  on  assumptions  about  the- 
effect  of  computer  downtime  on  training  time  which  seem  more  realistic  as 
determined  from  data  collected  for  this  study.  First  it  was  assumed  for 
downtimes  of  30  minutes  or  less  that  there  would  be  no  measurable  increase 
in  training  time.  For  downtimes  with  a  duration  between  30  minutes  and  60 
minutes,  training  time  would  be  extended  for  one-half  the  student  waiting 
time.  And,  finally,  for  downtimes  which  exceed  1  hour  there  would  be  an 
extension  of  training  time  equal  to  the  amount  of  time  all  students  spend 
waiting  for  the  computer.  The  equations  used  for  Case  II  are  as  follow: 


L  =  0:  If  D  <  1/2  Hour 

L  =  (1/2) (1/2) (C) (S) :  If  1/2  Hour  <  D  <  1  Hour 

L  =  (1/2) (C) (S) :  If  D  >1  Hour 

where:  D  =  Duration  of  Interruption. 


41 


r 


Technical  Report  119 


SECTION  VI 

SUMARY,  CONCLUSIONS,  AND  RECOMMENDATIONS 


SUMARY 

The  present  performance  of  the  CMI  system,  as  deduced  from  data 
collected  during  March  to  October  1981  showed  that  response  times  were 
satisfactory,  averaging  less  than  three  seconds  per  transaction,  and  the 
frequency  and  duration  of  interruptions  were  relatively  low.  For  most  days, 
the  system  was  available  95  percent  or  more  of  the  time  during  which  the 
schools  were  in  operation.  While  there  were  rather  severe  problems  in 
response  time  and  interruptions  prior  to  March,  improvements  in  system 
software  and  operating  procedures  have  significantly  reduced  these  problems. 
The  system  does  not  have  the  most  efficient  design  in  terms  of  disk  access 
and  other  system  software  and  potential  exists  for  upgrading  system  capacity 
by  implementing  some  design  change  in  these  areas. 

The  largest  single  use  of  the  present  CPU  is  MILPERSIS  followed  by  CMI 
and  then  the  maintenance  of  the  Mil SA  operating  system.  Together  these 
three  uses  account  for  85  percent  of  the  total  processing  time  which  is 
used.  However,  all  uses  together  do  not  fully  utilize  the  available 
capacity  although  there  are  periods  during  the  day  in  which  the  system  is 
nearly  fully  utilized.  Computer  Managed  Instruction  is  effectively  given 
precedence  over  all  other  users  through  allocation  of  CPU  time. 

Consequently,  at  the  present  level  of  utilization  the  non-CMI  users  do  not 
appear  to  significantly  impact  CMI  processing.  Even  during  those  periods  in 
which  the  capacity  is  being  fully  utilized,  CMI  is  allocated  all  the 
processing  time  required.  There  is  some  minor  contention  for  disk  access 
between  MILPERSIS  and  CMI  and  under  worst  case  conditions  CMI  performance 
degradation  is  estimated  at  5  to  10  percent.  It  is  expected,  however,  that 
continued  expansion  of  CMI  will  ultimately  force  delays  in  the  processing  of 
non-CMI  transactions.  As  previously  indicated,  the  level  of  loading  at 
which  delays  will  become  significant  can  only  be  determined  by  conjecture 
using  currently  available  data.  Development  and  maintenance  work  on  both 
the  CMI  and  non-CMI  software  and  hardware  is  usually  scheduled  during  off 
duty  hours.  Such  scheduling  reduces  the  impact  of  development  work  on  the 
effective  operation  of  the  CMI  system. 

CONCLUSIONS 

The  essential  and  most  relevant  management  problem  is  to  determine 
alternatives  which  are  both  technically  and  economically  efficient  to  enable 
the  system  to  maintain  performance  and  to  provide  for  future  growth.  Before 
any  meaningful  economic  analysis  can  be  performed,  it  will  be  necessary  to 
determine  at  what  point  increased  CMI  loading  will  cause  problems.  It  will 
then  be  necessary  to  determine  the  source  of  the  problems,  and  then  to 
evaluate  alternatives  for  overcoming  them. 

Factors  which  would  adversely  impact  the  operation  of  the  current  CMI 
system  could  only  be  postulated  and  cannot  be  deduced  or  observed  from  any 
degradation  in  performance  of  the  CMI  system  as  it  is  presently  operating. 
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File  access  procedures,  hardware  limitations,  system  design,  non-CMI  process¬ 
ing  requirements,  fluctuations  in  CMI  processing  requirements,  and  many 
other  factors  interact  in  such  dynamic  fashion  that  even  those  intimately 
familiar  with  the  existing  system  cannot  conclude  at  what  level  of  increased 
operation  significant  degradation  in  response  time  would  occur.  While  it 
is  reasonable  to  assume  that  the  system  will  become  overloaded  at  some 
increased  level  of  operation,  it  is  not  readily  apparent  what  hardware  or 
software  components  or  management  problems  will  prove  to  be  the  "weak"  link. 

Two  methods  can  be  used  to  determine  what  improvements  will  be  required 
in  order  to  maintain  system  performance  as  more  students  are  placed  under 
the  CMI  system.  First,  a  comprehensive  simulation  model  would  provide  one 
means  of  determining  the  system  limitations  and  enable  queries  of  the  "what 
if"  type  relating  to  system  upgrading.  Simulation  could  also  be  used  to 
determine  if,  and  to  what  extent,  system  performance  will  be  degraded  by 
demands  for  service  from  non-CMI  users  such  as  MILPERSIS.  Various  alter¬ 
native  improvements  could  be  modeled  and  costed  and  the  most  cost-effective 
alternative  identified.  A  generic  CMI  simulation  model  is  currently  being 
developed  which  may  be  used  to  support  studies  of  this  type  for  system 
expansion. 

Second,  the  student  load  on  the  present  system  could  be  carefully  and 
gradually  increased  until  significant  performance  degradation  becomes  appar¬ 
ent.  There  is  at  present  no  reliable  way,  either  objectively  or  subjectively, 
to  determine  at  what  level  of  student  loading  such  performance  degradation 
will  occur.  However,  as  performance  degradation  occurs,  as  it  inevitably 
will,  corrective  action  can  be  taken  to  maintain  system  performance.  Such 
corrective  actions  must  be  technically  feasible  and  should  be  economically 
efficient.  There  are  indeed  certain  risks  associated  with  the  continued 
expansion  of  the  present  CMI  system,  but  these  risks  appear  minimal.  The 
most  serious  problem  would  be  complete  and  prolonged  system  failure  result¬ 
ing  from  overloading  which  appears  to  have  a  low  probability  of  occurring. 

The  more  likely  impact  of  overloading  is  a  steady  increase  in  response  time. 

A  reasonable  increase  over  the  existing  response  time  of  two  to  three  seconds 
could  be  tolerated  without  having  a  significant  impact  on  training  time 
since  system  specifications  call  for  a  response  time  of  30  seconds  or  less. 
This  would  indicate  that  current  response  time  could  be  increased  without 
system  specifications  being  exceeded. 

It  is  inevitable  that  at  some  point  computer  downtime  will  affect  train¬ 
ing  time.  However,  the  present  study  was  concerned  with  both  duration  of 
downtime  as  well  as  the  frequency  of  interruptions  and  to  what  extent  these 
have  impacted  on  training  time.  Available  data  does  not  suggest  a  direct 
one  to  one  relationship  between  computer  downtime  and  length  of  student 
time  in  training.  In  fact,  the  data  available  suggests  that  short  (30 
minutes  or  less)  and  relatively  infrequent  downtimes  have  a  minimal  impact 
on  student  learning  for  a  CMI  system.  Student  time  to  mastery  is  related 
to  time  spent  in  fulfilling  certain  specified  conditions  of  learning.  Most 
of  the  student's  studying  and  learning  experiences  in  fulfilling  those 
conditions  do  not  involve  the  computer.  Computer  downtime  could  only  affect 
the  learning  rate  of  those  students  who  demand  service  and  the  number  of 
students  demanding  service  will  depend  on  the  length  of  downtime  and  frequency 
of  student  interaction  with  the  computer. 
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Most  learning  centers  do  not  have  a  complete  manual  back-up  system. 

However,  where  such  systems  exist,  they  appear  to  be  very  effective  in  mini¬ 
mizing  negative  effects  of  computer  downtime.  Properly  designed  back-up 
systems  could  be  developed  which  would  be  effective  in  managing  prolonged 
downtime.  The  high  availability  of  the  CMI  systems  makes  it  questionable 
whether  the  development  and  maintenance  of  full-scale  back-up  systems  would 
be  cost  effective.  Most  of  the  school  staff  consulted  during  this  study 
were  of  the  opinion  that  there  was  no  measurable  impact  on  training  time 
from  the  relatively  minor  interruptions  which  have  occurred  during  the  latter 
part  of  the  time  period  of  this  study.  However,  staff  estimates  of  how 
long  the  computer  must  be  down  before  there  would  be  a  significant  exten¬ 
sion  of  training  time  varied  from  15  minutes  to  1  hour.  From  the  above 
analysis  it  is  estimated  that  the  computer  would,  in  most  cases,  need  to  be 
down  for  an  hour  or  more  before  training  time  would  be  extended  significantly. 
Between  March  and  October  1981  there  were  relatively  few  instances  where 
the  CMI  service  to  a  course  was  down  for  an  hour  or  more  during  the  entire 
shift.  Approximatley  95  percent  of  the  interruptions  lasted  30  minutes  or 
less. 

It  is  reasonable  to  assume  that  a  degree  of  distributed  processing  should 
be  considered  for  any  major  expansion  of  the  current  CMI  system.  Possible 
advantages  of  distributed  processing  include  elimination  of  the  need  for 
communication  lines  to  process  student  transactions,  the  improvement  of 
local  command  control  and  service,  and  the  provision  of  redundancy  where 
possible.  Disadvantages  include  the  potential  loss  of  central  control, 
possible  scale  diseconomies,  and  increased  difficulty  in  maintaining  soft¬ 
ware  and  hardware  standardization.  The  cost  effectiveness  of  distributed 
processing  will  depend,  in  part,  on  the  costs  of  upgrading  the  existing 
system  as  well  as  development  and  implementation  costs  of  a  distributed 
system.  More  data  needs  to  be  collected  to  determine  the  requirements  and 
costs  of  upgrading  the  present  system.  Simulation  would  provide  a  basis 
for  obtaining  these  data.  At  this  time,  there  is  not  an  acute  need  for  a 
major  redesign  of  the  present  system.  Subject  to  a  rapid  and  unexpected 
increase  in  courses  placed  in  the  CMI  system,  time  is  available  to  make  a 
complete  assessment  of  system  need  and  to  formulate  a  conceptual  system 
design  prior  to  any  new  development  and  implementation.  Additionally,  proto¬ 
type  implementation  and  evaluation  are  recommended  before  considering  wide- 
scale  application.  A  site  phasing  implementation  plan  should  be  developed 
which  would  assure  training  continuity  during  distributed  system  integration. 

As  stated  previously,  the  existing  system  is  a  good  one  and  it  can  be 
expanded.  Every  possible  step  should  be  taken  to  assure  that  a  replacement 
system  will  perform  more  efficiently.  It  should  also  be  noted  that  a  replace¬ 
ment  system  will  form  the  basis  for  Navy  computer  based  instructional  manage¬ 
ment  during  the  next  decade.  If  the  concept  formulation  phase  of  this  develop¬ 
ment  does  not  include  life  cycle  cost  benefit  assessment  and  if  state-of- 
the-art  network/communications/data  base  architectures  are  not  considered, 
there  is  a  high  probability  of  serious  consequences  for  Navy  training.  The 
training  community  may  be  forced  to  live  with  a  deficient  system. 

Distributed  processing  is  not  a  fixed  approach  to  data  processing  but 
can  encompass  a  wide  range  of  software  and  hardware  configurations.  Dis¬ 
tributed  processing  also  requires  decisions  about  which  functions  can  most 
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effectively  or  efficiently  be  processed  remotely  at  the  school  rather  than 
at  a  centrally  located  facility.  Consequently,  there  can  be  a  large  number 
of  alternative  systems  defined  as  distributed  processing  which  will  be  cap¬ 
able  of  providing  the  required  future  CMI  services.  A  number  of  the  tech¬ 
nically  feasible  alternatives  should  be  evaluated  in  order  that  the  most 
cost-effective  system  can  be  identified  for  implementation. 

A  final  and  obvious  conclusion  is  that  if  the  CMI  processing  requirements 
continue  to  increase,  the  CMI  system  will  eventually  be  overloaded  and  require 
upgrading.  At  least  two  important  areas  must  be  addressed  to  determine  the 
most  cost-effective  way  to  upgrade  the  system.  First,  the  maximum  efficient 
capability  of  the  present  system  is  unknown.  Presently,  there  is  no  reliable 
way  to  identify  the  potential  difficulties  which  will  be  encountered  with 
increased  processing  requirements;  therefore,  there  is  no  reliable  way  to 
determine  the  marginal  cost  o*  upgrading  the  present  system.  Simulation  or 
future  experience  gained  from  increasing  the  load  on  the  present  system 
will  eventually  provide  answers  to  this  problem.  Second,  at  what  point  the 
processing  requirements  will  exceed  the  capability  of  the  present  system 
depends  on  the  rate  and  processing  requirements  of  new  courses  brought  under 
CMI.  The  processing  requirements  may  not  increase  as  rapidly  for  group- 
paced  courses  requiring  only  testing  support  as  they  would  for  self-paced 
courses. 

RECOWENDATIONS 

1.  Develop  a  comprehensive  simulation  model  which  will  provide  a  capa¬ 
bility  for  determining  "bottlenecks"  in  t.he  present  system  and  for  evaluating 
alternative  CMI  expansion  strategies.  Su  h  a  model  would  provide  insight 
into  the  efficiency  and  effectiveness  of  a.ternative  expansion  strategies. 

2.  Develop,  implement,  and  evaluate  a  prototype  distributed  processing 
system.  Such  a  system  should  be  ready  for  operational  implementation  when 
and  if  it  becomes  noneconomical  to  further  expand  the  present  system. 

3.  Using  data  from  a  planned  CMI  course  implementation  schedule,  the 
simulation  model,  results  from  the  prototype  distributing  system,  and  the 
technical  capabilities  of  microcomputer  technology,  develop  a  long-range 
plan  for  system  expansion.  Options  should  include  expanding  the  present 
CMI  system,  implementing  distributed  processing,  and  viable  options  which 
utilize  both  approaches. 

4.  Develop  a  workable  strategy  for  managing  the  students  during  computer 
downtime  with  emphasis  on  short  interval  interruptions. 

5.  Because  of  high  system  reliability,  reevaluate  the  cost-effective¬ 
ness  of  existing  requirements  for  developing  and  maintaining  comprehensive 
manual  back-up  systems  to  manage  students  during  the  longer  downtimes. 
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Figure  A-2.  Mean  CMI  Response  Time  by  Day  of  Week 
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Figure  A-4.  Total  Interruptions  by  Day  of  Week 
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Figure  A-5.  Histogram  of  Mean  Downtime  by  Day  of  Week 
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Figure  A-6.  Total  Time  System  Reported  Down 
for  Each  Day  of  Week 
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Figure  A-8.  Total  Time  System  Reported  Down 
for  Lach  Location 
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Figure  A-9.  Mean  Percentage  of  Time  System  Not 
Available  at  Each  Location 
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